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Operational Test and Evaluation Force (OPTEVFOR). 
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EXECUTIVE SUMMARY 

In the 1970-71 time period. President Nixon's Blue Ribbon 
Defense Panel (BRDP) and the U.S. Congress became deeply involved 
in DOD weapon system test and evaluation (T&E) policy and inde¬ 
pendence. The independence they addressed was separation of the 
system tester from the developer and user, both of which they 
considered system advocates. Further, neither Washington group 
felt that sufficient testing was being accomplished prior to key 
acquisition decision points. As a result of this high level 
criticism, OSD directed the three Services to establish independ¬ 
ent test agencies (ITA's) and to conduct more and improved T&E. 

The purpose of this study was to learn more about the causes 
for and results of that OSD direction. The review included a 
survey of official government records such as the BRDP Report and 
sources containing Congressional testimonies. Significant excerpts 
from these documents can be found in Section II. 

The Services established their ITA's during the period 1972- 
1974. Section III reports on the agencies themselves, including 
such factors as operational dates, organization structure and 
manning. Comparisons and contracts are made between the Army, 

Navy and Air Force ITA's. In addition, the current mission, 
number of major/non-major Lest programs and OT&E management poli¬ 
cies of each agency are reviewed. 
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One initial and major objective of this project was to in~ 
vestigate and compare costs of OT&E before and after the intro¬ 
duction of ITA's. However, this objective was modified in "mid¬ 
stream” when it became apparent that actual T&E costs are just 
not available. Section IV addresses this cost accountability 
problem and provides a look at current ITA budgeting and funding. 
In addition, it introduces some of the non-dollar costs of oper¬ 
ating under the ITA concept. 

Sections V and VI summarize the study and conclude that 
current OT&E costs are very high. There is evidence of "over- 
testing” and loss of program control by the DOD Program Manager. 

Three recommendations pertinent to reducing costs of OT&E 
are presented in Section VII. 
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In this Bicentennial year whan all Americans are thinking 
of and frequently hearing the word "independence"/ it seemed be¬ 
fitting to address a recent major change to the U.S. military's 
systems acquisition process - Independent Test Agencies (ITA's). 

Over the past half decade, much has been written, stated 
and directed on the need for arid benefits of independent testing 
in the DOD systems acquisition process. Since the July 1970 
Blue Ribbon Defense Panel Report, Test and Evaluation (T&E) poli¬ 
cies of the military services have been significantly altered. 
Separate independent test agencies (ITA's) have been established 
by the Army, Navy and Air Force. These agencies, although organ- 
ized somewhat differently in their individual Service's hierarchy, 
have one fundamental element in common; each maintains its testing 
independence from both the developer and the user, reporting 
directly to Service headquarters. The three ITA's vary consider¬ 
ably in size from the Air Force's organization of approximately 
230 people to ha Navy's OPTEVFOR which numbers over 1,400 offi¬ 
cers, enlisted and civilian personnel. 

It is extremely dirficult to take issue with the theory that 
independent testing and reporting provides increased objectivity 


i 




to the systems acquisition process. Likewise, one would be 
foolish to disagree with the fact that objective testing and 
evaluation are essential to proper decision making throughout 
all phases of systems acquisition. The point in the process 
where test results appear to be most critical (in terms of dol¬ 
lar decisions and high level review) is at Defense System Acqui¬ 
sition Review Council (DSARC) III. It is from this review that 
the Secretary of Defense determines whether to approve or deny 
production go-ahead for the system. This decision, which nor¬ 
mally follows the completion of full-scale development opera¬ 
tional testing, draws heavily upon the demonstrated system per¬ 
formance data as reflected in the reports by the ITA's. These 
agencies exist today primarily because of the significance of 
the DSARC III decision in terms of the nation’s defense posture 
and budget. 

Because two of the three service ITA's are relatively young 
organizations and the third has just recently incorporated inde¬ 
pendent operational testing, their contributions to date are 
difficult to measure. No "track records" are available since 
no weapon systems have yet been deployed for which the ITA's 
have had full test management responsibility. Future analysis 
of their worth should address items such as; Are the systems 
being fielded by the services more dependable, better performing. 
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and/or less costly to operate and maintain? If they are, how 
much of the increased success can be attributed directly or 
indirectly to the activities of the ITA's? And finally, what 
has been the cost of the increased :'ystem effectiveness due to 
ITA involvement? 

OSD, when directinq Service ITA's, was convinced that in¬ 
dependent testing was essential to the decision-mciking processes 
of weapon systems acquisition. 

This paper reviews the issues and activities leading to 
that direction, discusses how each service implemented that dir¬ 
ection, and addresses some of che resulting costs. The costs 
examined are not limited to those items that can fcfe identified 
on a budget or other financial document. Just as important are^ 
those factors more difficult for a pr'.gram manager to quantify 
such as system responsibility, program control and schedule. 
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SECTION II 

BACKGROUND 

Previous Operational Testing 

Until quite recently, operational testing of new weapon and 
other military systems was conducted and reported on by the im¬ 
plementing (developer) and using commands. Now and into the 
foreseeable future, separate agencies within each service are and 
will be responsible for managing or monitoring all operational 
testing. 


DT&E versus OT&E 


Before proceeding with the causes for this drastic change 
in T&E policy and procedure, a brief differentiation between 
operational testing and development testing is in order. Devel¬ 
opment test and evaluation (DT&£) is concerned primarily with 
the engineering function of the system design. 

DT&E is conducted to verify accomplishment of 
the development objectives; these include refining 
engineering design, verifying that design risks have 
been minimized, demonstrating that the system will 
meet specifications, and estimating the military 
utility. DT&E is essentially a detailed engineering 
analysis of a system's performance, beginning with 
individual subsystems and progressing through a com¬ 
plete system, where system design is tested and 
evaluated against engineering and operational criteria 
by the implementing command... (1) 
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On the other hand, operational test and evaluation (OT&E) focuses 
on the intended use of the system in its operational etivironment. 

OT&E is conducted to estimate the system's 
military utility, operational effectiveness, operational 
suitability, and to identify deficiencies. In addition, 

OT&E provides information on organization, personnel 
reo(uitements, doctrine, and tactics. OT&E, normally 
composed of initial OT&E (lOT&E) and follow-on OT&E 
(FOT&E), is essentially an operational analysis of 
a system's performance where the complete system is 
tested and evaluated against operational criteria by 
personnel with the same qualifications as those who 
will use, maintain and support the system when deployed. (1) 


What Prompted ITA's? 

During the period 1950-1970, defense system planners and 
developers received "little flak" from higher levels of the 
government bureaucracy on the methods being used to develop and 
test new systems.. Until then, most controversy surrounded the 
actual need for the system in the Defense inventory and the cost 
of putting it there. Concern over whether a system was actually 
ready for production or performed to certain specifications was 
minimal outside DOD. This lack of concern is attributed to a 
general satisfaction vdthin congressional and executive arenas 
that government Research, Development, Test and Evaluation (RDT&E) 
activities had been successful in maintaining U.S. military 
superiority, in winning the ICBM and space race and putting man 
on the moon. In addition, there were factors such as the Vietnam 
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Conflict which had caused DOD to defer many major weapon system 
acquisition programs. As the 1970's approached, economic condi¬ 
tions and domestic issues such as social programs and environ¬ 
mental improvement caused Congress to intensify its scrutiny of 
DOD expenditures, especially in the area of major weapon systems 
acquisition. As a result of this intensification on the part of 
Congress and the President, T&E policies of the military services 
would change dramatically. 

The July 1970 Blue Ribbon Defense Panel Report (also known 
as the Fit 2 hugh Report) was the driving force behind this change. 
The panel, formed by President Nixon to examine the Department of 
Defense, made a searching review of DOD systems acquisition 
policies, including T&E. The following critical judgment con¬ 
cerned DOD Operational Test and Evaluation (OT&E) activities: 

There has been an increasing desire, particularly 
at OSD level, to use data from OT&E to assist in the 
decision-making process. Unquestionably, it would be 
extremely useful to replace or support critical assump¬ 
tions and educated guesses with quantitative data ob¬ 
tained from realistic and relevant operational testing. 

Unfortunately, it has been almost impossible to 
obtain results which are directly applicable to decision 
or useful for analysis. Often, test data do not exist. 

When they do, they frequently are derived from tests 
which were poorly designed or conducted under insuffi¬ 
ciently controlled conditions to permit valid compari¬ 
sons. It is especially difficult to obtain test data 
in time to assist in decision-making. Significant 
changes are essential if OT&E is to realize its poten¬ 
tial for contributing to important decisions, particularly 
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where the tests and the decisions must cross Service 
lines. (2) 

The pressure continued. Witnesses appearing before 1971 Senate 
and House Conunittees testified that: 

...funds for testing and evaluation were 
inadequate, and organizational responsibility was 
fragmented. At all levels of management within 
DOD, testers were not independent of the devel¬ 
opers. (3) 

Mr. G. W. Fitzhugh, Chairman of the Blue Ribbon Defense Panel 
(B.RDP), addressing the operational testing problem at 1971 
hearings before the Senate Committee on Armed Services, stated: 

One of the most urgent needs for improvement of 
the entire weapon system acquisition process is more 
effective operational test and evaluation. 

Everyone seems to agree that operational test 
and evaluation (OT&E) is very important; however, 
there are significant differences of opinion as to 
what it encompasses, what its proper objectives are., 
and what organizations and methods are necessary to 
accomplish it most effectively. (4) 


Informal Direction 

The driving force within DOD for changing T&E policies was 
Secretary Packard, Deputy Secretary of Defense under Melvin 
Laird. As a result of external studies and his own investiga¬ 
tions, Secretary Packard issued a series of memoranda which 
directed expedited change to T&E procedures. A memorandum, 
dated 11 February 1971, directed the Services to arrange for 

OT&E to be managed and conducted by an agency separate from the 
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developing command and which reports the results of all opera¬ 
tional test and evaluation directly to the Chief of the Service. 
The memo also directed the establishment of aii OT&E focal point 
within the Service headquarters. In the same document, Mr. 
Packard established a Deputy Director, Defense Research and 
Engineering, to be the OSD focal point for T&E matters. (5) 


Formal Direction 

On 13 July 1971, the newly published DOD Directive 5000.1 
officially tied OT&E to the DSARC III decision point. 

Test and evaluation shall commence as early as 
possible. A determination of operational suitabili..,y, 
including logistic support requirements, will be made 
prior to large scale production commitments, making 
use of the most realistic test environment possible 
and the best representa+-ion of the future operational 
system available. The results of this operational 
testing will be evaluaced and presented to the DSARC 
at the time of the production decision. (6) 

Department of Defense Directive 5000,3, Test and Evaluation, 

dated January 19, 1973, provided additional and more specific 

T&E policy guidance than DODD 5000.1. 

In each DOD Component there will be one major 
field agency separate and distinct from the develop¬ 
ing/procuring command and from the using command which 
will be responsible for OT&E and which will; 

a. Report the results of its independent 
test and evaluation directly to the 
Military Service Chief or Defense Agency 
Director. 

b. Recommend directly to its Military Service 
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Chief or Defense Agency Director 
the accomplishment of adequate OT&E. 
c. Insure that the OT&E is effectively 
planned and conducted. (7) 

Official OSD direction for ITA's had been published 


received and understood. 




SECTION III 


SERVICES ESTABLISH ITA's 

Following the release of the Blue Ribbon Defense Panel 
Report in mid-1970, the Services took a more critical look at 
their past and present RDT&E activities with special emphasis 
on OT&E. Various studies were conducted based upon internal 
criticisms as well as those inefficiencies cited in the BRDP 
Report and several Concressional Hearing documents. 

The Navy 

The Navy's problem of coming up with a test agency independ 
ent of the developer and user was easier to solve than those of 
its sister Services. OPTEVFOR, which is headquartered in 
Norfolk, Virginia, had been in existence since 1944 when it was 
established for the purpose of studying the Japanese "Kamikaze" 
threat. Following World War II, its mission had been expanded 
in the area of tactics evaluation and by 1970 it was also con¬ 
ducting and supporting development agency and user testing. 
Navy-wide. To achieve the desired independence identified in 
the BRDP Report, meant basically a shift in emphasis to earlier 
program involvement by OPTEVFOR. Tnis change in miss.on orienta 
tion was accomplished in the late-1971 to mid-1972 time period. 

WJiile the reorientation did not involve an appreciable organi- 
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zation restructuring or increased manpower allocations, it did 
require a different and more expensive mix of personnel. The 
Navy’s "hands on" operational testing concept demanded experi¬ 
enced operational officers cind high3.y skilled NCO's familiar 
with current mission, tactics, policies and procedures. OPTEVFOR's 
current personnel strength totals 1,404 authorizations which 
includes 310 officers, 868 enlisted, and 45 Department of the Navy 
civilians. 315 positions of the 1,404 authorizations are located 
at OPTEVFOR Headquarters. The remainder are spread across sub¬ 
ordinate commands, aircraft squadrons and detachments (see Figure 
3-1). (8) 

OPTEVFOR's stated mission in the area of operational testing 
is to plan, conduct and report on OT&E for all Navy acquisition 
programs. COMOPTEVFOR reports OT&E results directly to the Chief 
of Naval Operations (CNO) and provides Developing Agencies (DA) 
with all significant OT&E test data and analyses. OPTEVFOR is 
also responsible for coordinating/conducting joint T&E and oper¬ 
ational testing for the USMC. Independent evaluation reports on 
USMC T&E are made directly to Commandant of the Marine Corps and 
to the CNO. (9) 

OPTEVFOR is currently managing OT&E for 219 systems and is 
responsible for monitoring OT&E on an additional 209 CNO assigned 
projects. Furthermore, this ITA is providing DT&E assistance 
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to the DA*s on approximately 150 development efforts. (10) 


The Army 

Pursuant to a letter from The Adjutant General, Department 
of the Army, 25 September 1972, the United States Army Test and 
Evaluation Agency (OTEA) was established effective that date at 
Fort Belvoir, Virginia. It was activated under the jurisdiction 
of the Assistant Chief of Staff for Force Development (ACSFOR) 
and a TDA was established with a personnel authorization of 120 
spaces, which included 53 officers, 2 enlisted men, and 65 
civilians. (11) 

In 1973, OTEA was reorganized with a personnel authorization 
of 200 spaces made up of 103 officers, 19 enlisted, and 78 civil¬ 
ians. Changes in 1974 made OTEA a field operating agency under 
the Chief of Staff, Army, and increased total strength to 223. 

The most recent reorganization took place effective 15 March 1975, 
when strength was again adjusted upward to OTEA's present manning 
of 250 spaces, which includes 125 officers, 20 enlisted men, and 
105 civilians. 

The stated mission of OTEA is to support the material acqui¬ 
sition and force development processes by exercising responsi¬ 
bility for all operational testing and by managing force develop¬ 
ment testing (FDTE) and joint user testing for the Army. (12) 
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The OTEA organization includes a command group, a coordinating 
staff and a functional staff (see Figure 3-2). 

OTEA is currently managing OT&E for 22 major and 17 Category 
I non-major systems. In addition, the agency is monitoring 
operational test activities on more than 250 lesser category 
non-major systems. Out of the pure OT&E environment, OTEA is 
involved in three major FDTE efforts and nine joint user testing 
exercises with other Services. (13) 

The Air Force 

The Air Force was last to formally establish their OSD 
directed ITA, doing so on 1 January 1974. Where the Array and 
Navy had earlier complied explicitly v;ith the BRDP recommenda¬ 
tion for separate, independent OT&E agencies (OTEA and OPTEVFOR), 
the Air Force initially attempted to achieve independence by 
placing full OT&E responsibility on the functional using com¬ 
mands (TAG, SAC, etc.). While this complied with the letter of 
the BRDP recommendations and in practice was a step in the right 
direction, it fell short of the intent and therefore was suspect 
and often subject to criticism. The Air Force designated its 
ITA the Air Force Test and Evaluatic' Center (AFTEC), located 
this field command agency at Kirtland AFB, New Mexico, with 
direct reporting to the Chief of Staff, Air Force. 
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since becoming completely operational in October 1974, 
AFTEC's manning has increased from an initial authorization of 
208 to a present strength of 231. This includes 155 officers, 

32 enlisted personnel and 44 civilians. AFTSC manning is pro¬ 
jected to increase to 290 positions within the next year. 

The stated mission of AFTEC is to manage the Air Force 
operational test and evaluation program and assess the military 
utility and operational effectiveness and suitability of major 
and designated non-major systems. The center will also actively 
monitor all non-major systems OT&E conducted by the major com¬ 
mands through review and approval of test plans and review of 
and comment on test reports. In its role as the jT for 

Air Force OT&E, AFTEC will provide policy recommendations for 
Hq. USAF approval and subsequent implementation by all major air 
commands. AFTEC will act as the Air Force spokesman to the 
DSARC on matters pertaining to OT&E and will support Congres¬ 
sional requests for OT&E information on weapon systems for which 
procurement funds are being requested. (14) The AFTEC organi¬ 
zation, similar to the Army's OTEA, includes a command section 
and coordinating and functional staff elements (see Figure 3-3). 

AFTEC is currently managing OT&E for 47 major and designated 
non-major programs. In addition, at the present time the center 
is monitoring 144 major command operational test efforts and is 
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involved in nine OSD-directed joint Service tests. (15) 


OSD Officials Coniment 


Hence, the Services achieved independence in systems 
acquisition testing by late 1974. In a statement to Congress 
in March 1975, Lt. General Starbird, USA (Ret), then Deputy 
Director (T&E), DDR&E, remarked: 

Each of the three Services now has a strong, 
mature and effective organization for the accomplish¬ 
ment of independent operational test and evaluation. 

(They are the Army's Operational Test and Evaluation 
Agency, the Navy's Operational Test and Evaluation 
Force, and the Air Force's Test and Evaluation Center.) 
(16) 

More recently, 3 February 1976, Dr. Malcom Currie, Direct 
of Defense Research and Engineering, told Congress: 

The past few years have seen significant im¬ 
provements in DOD procedures for conducting T&E of 
major weapon systems. High level emphasis on T&E was 
assured initially by creation of the Office of Deputy 
Director of Defense Research and Engineering (Test 
and Evaluation). First under the direction of Lieuten¬ 
ant General Alfred D. Starbird, USA (Ret.) and now under 
Lieutenant General Walter E. Lotz, Jr., USA (Ret.), this 
office has closely monitored all major, and many less- 
than-major, weapons acquisition programs. 

Further strengthening of DOD T&E has been 
achieved by the establishment of independent opera¬ 
tional T&E agencies within each of the Services. These 
are the Navy's Operational Test a.i Evaluation Force, 
the Army's Operational and Test Agency and, most re¬ 
cently, the Air Force's Test and Evaluation Center. 

These agencies plan and conduct tests and submit their 
independent evaluations directly to the Chiefs of thier 
respective Services. Their evaluations provide the 
objective assessment of the operational effectiveness 
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of new weapon systems which is essential to proper 
functioning of the weapons acquisition process. 
Creation of these agencies has had a significant 
impact on systems development and procurement as 
evidenced by the greater attention being directfe.d 
to operational effectiveness within the Services 
and by the greatly improved quality of T&E data 
now being presented to the Defense Systems Acquisi¬ 
tion Review Council (DSARC). 

A significant fallout from the emphasis on 
T&E is a better definition of weapon system perform¬ 
ance goals. The T&E agencies of the Services and 
the DD (T&E) are insuring that DCP's specify all 
essential performance characteristics and minimum 
acceptable values for these parameters in advance 
of undertaking tests. This requires an explicit 
definition of the design parameters in operational 
terms early in the acquisition cycle. Such defini¬ 
tion not only provides a firm basis for planning 
the type and extent of testing but also provides a 
clear basis for the evaluation of the military 
utility of a new system. (17) 


16 






SECTION IV 


ITA'S - SOME OF THE COSTS 

Identilying Dollar Costs 

Accurate identification of the dollar costs associated with 
the implementation and operation of the current system of in¬ 
dependent testing (ITA's) is extremely difficult. It would be 
nice to be able to report that under the old policy, a system 
OT&E cost "X" dollars, while an OT&E of a comparable system to¬ 
day costs "Y" dollars. One reason for the difficulty is that all 
Services are still involved in somewhat of a transition period 
between the old and new procedures. However, the primary reason 
lies in the fact that T&E costs under the previous policy of 
joint developer/user OT&E are not that well documented. In other 
words, no firm baseline for comparison exists. The various OT&E 
costs were borne by different commands, different agencies within 
the commands, and utilizing different sources and types of funds. 
OT&E dollars are especially difficul*: to extract from development 
contracts with industry where costs are "hidden" in labor (e.g., 
direct and indirect), materials (e.g., prime system components, 
spares and test equipment) and overhead. Test range costs in¬ 
volve both military and commercial contractor support expenses. 
Significant T&E dollars are expended by supporting and partici- 











pating organizations such as the using, maintaining, logistics 
and training commands. These costs are incurred over relatively 
long periods of time, beginning during the test planning phase 
and extending through actual systems testing and test report 
submission. 

Under the new policy where a single independent agency is 
responsible for total management of OT&E, cost accounting should 
be a more manageable function. However, there is little indica¬ 
tion at this time of any appreciable change. The true cost of 
OT&E remains to be somewhat of an unknown. Tnere are, however, 
some facts and figures indicating that the ITA's do represent an 
increase in the cost of RDT&E. 

The manpower of these new agencies is one very evident cost 
factor. People costs, since they currently account for over 50% 
of the national defense budget, must be considered when addressing 
the price of independence. There have been few manning trade¬ 
offs as a result of ITA management of OT&E. The current person¬ 
nel strength of test functions within program ofrices, test 
ranges and using/supporting command elements do not appear to 
have been affected. In fact, from a systems program office view¬ 
point, additional manning might be justified on the basis that 
the ITA represents another agency for coordination and/or ap¬ 
proval on all <-jSt related activities. 
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When OTEA and AFTEC were being established, the majority of 
the new personnel authorizations were to be achieved by a like 
reduction of authorized positions in the other commands. How¬ 
ever, most of the manpower authorization trades resulted in 
units giving up the more expendable positions. The result of 
this was that many position conversions were required. For ex¬ 
ample.- administrative and clerical positions had to be converted 
to laboratory and other scientific specialties. Likewise, many 
generalized enlisted positions were "given up” that required 
coriversion to specialized officer authorizations. As has been 
previously mentioned, the Navy’s OPTEVFOR OT&E structure re¬ 
quired more senior and more experienced officer capabilities. 
Also, as previously indicated, there have been manpower increases 
for both OTEA and AFTEC since they were established. Three 
capably staffed OT&E organizations now exist but at a significant 
cost, personnel-wise. 

In the case of the Army and Air Force, facilities for the 
new organizations had to be located. Once acquired, f'e attend¬ 
ant costs of outfitting and maintaining these facilities is a 
consideration. 

The annual funds budgeted for and managed by the ITA's 
are as indicated in the following table: 
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TABLE 4-1 


ITA FUNDING ($ M) 

OPTEVFOR OTEA AFTEC 



FY 76 

FY 77 

FY 76 

FY 77 

FY 76 

FY 77 

O&M 

.57* 

.64* 

5.20 

7.39 

5.67 

5.69 

RDT&E 

3.60** 

4.00** 

3.90 

5.44 

1.64** 

5.64 


*Hqs. only (Does not include A/C Squadrons & Detachments) 

**Does not include project RDT&E funds. 

Note: Table 4-1 does not re/itjt procurement or RDT&E 

costs for test equipm -..w a:?d unique instrumentation 
requirements (these cost.: are substantial - for 
example, OTEA's estimate is $7-10 Million annually) 


Overtesting? 

A trend which is supported by many officials involved at all 
levels in the systems acquisition process is that of "more test¬ 
ing". This is particularly true in the areci of OT&E. However, 
it is difficult to substantiate that this increased testing is 
due solely to the existence of ITA's. The extreme emphasis on 
testing prior to major OSD decisions has been applied since 1970 
at every level across DOD. Additional systems testing is expen¬ 
sive and if it has, in fact, reached the point of "overtesting", 
dollars are being wasted. There are many military development 
command and certain OSD officials who believe and have stated 
that too much OT&E is now being conducted. One high OSD official 


recently remarked that, "it appears that the pendulum of opera- 
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tional testing has indeed swung too far". (18) There were 
many reports of systems overtesting. He continued that, "this 
overtesting results in an obvious waste of dollars and people 
when what is really needed is better conceived test plans". The 
latter would result in more effective system evaluation and at 
the same time save critical dollar and people resources. (18) 
Overtesting not only impacts a systems program in dollar 
costs. It is quite obvious that more testing normally involves 
the procurement of additional test items and other materials. 
Jikewise, increased testing clearly involves more time which 
equates to additional test range, using/supporting command and 
contractor personnel costs. However, the well used expression 
"time is money" does not go quite far enough for the systems 
program manager/director. Since he is the key individual in 
systems development, and totally responsible for a program from 
concept through operational deployment, "time is more than 
money.'" The program manager has a commitment to the user to 
provide a fully tested weapon system by specific need times 
referred to as Initial Operational Capability (IOC) and Full 
Operational Capability (FOC) dates. The user in turn has a 
commitment to the Service dnd the Service to the U.S. citizen 
in terms of providing adequate national defense. The remainder 


of this section will discuss some of these non-dollar co^^ts of 







ITA‘s as seen through the eyes of a DOD program manager (PM). 

Non-Dollar Costs 

The introduction of ITA*s into PDT&E has taken from the 
PM a significant portion of his control of the program. He still 
has total program responsibility, but no longer has the authority 
with which to control a major function of development, that of 
OT&E. He has lost the flexibility he once possessed through 
OT&E planning and budgeting. Trade-offs between testing and 
other phases of system acquisition can no longer be made within 
the program management office (PMO). 

The ITA also represents another agency with which the PMO 
must interface. In the current transition period, it has been 
especially difficult and time consuming to bring the iTA's "up 
to speed" in order that they possess the necessary knowledge and 
capability to assume full OT&E management. For future programs, 
this problem should be minimal since the ITA would have been "on 
board" since the conceptual phase of the programs. However, the 
interfacing requirements will continue to be significant since 
the ITA represents not only another coordination agency, but also 
an action agency on all test plans and major decision documents 
that impact the DSARC reviews. 

The existence of the ITA in its position of OT&E test manag¬ 
er has also resulted in the constriction of information flow 








between the PMO and the user; said another way, between the 
developer and the customer. The PM regards this loss of impor¬ 
tant dialogue as another cost of control since the critical 
feedback loop has been severed or at least interrupted. A like 
situation has developed between the PMO and the actual test 
agency, be it a government or Service test range/facility. 

The FM's of lesser priority programs feel that they stand 
to lose the most control due to the large, and what appears on 
the surface to be an unwieldy, number of programs being managed 
r'r monitored by the ITA's. They anticipate that the higher 
priority programs will receive more attention and by more senior 
and more experienced test managers. 

Probably the most significant cost of control, or loss of 
PM control, exists because of the location of the ITA's in the 
military hierarchy. These agencies report directly to the 
Service Chiefs. In addition, and as one ITA mission statement 
reflects, "AFTEC is wbe official Air Force spokesman to the 
DSARC...and supports Congressional requests...on weapon systems 
for which procurement funds are being requested". (15) This 
represents an unprecedented loss of program control from the PM 
to what is in fact a higher level agency. In -essence, the ITA 
activity can be regarded as a significant step backward in OSD's 
long-standing policy of decentralizing systems acquisition 
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management. 

The costs included in this section by no means form a 
complete list. They do, however, represent areas that should 
be addressed when and if a formal reassessment of ITA's is 
accomplished. 
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SECTION V 


SUMMARY 

During the 1970-71 time period, a President's panel and 
Congress expressed the need for additional test and evaluation 
prior to DOD major decisions on weapon systems acquisitions. 

The need was further defined as being twofold in nature. First, 
there was a requirement for an increased quantity of system 
testing and associated test data. And secondly, there was a 
need for more objective testing by separating the tester from 
the developer and user. In response, OSD directed 'DODD 5000.1 
and 5000.3) the Services to establish independent test agencies 
to manage all weapon systems OT&E. In addition, it was directed 
that these agencies report OT&E test results directly to the 
Service Chiefs. 

By 1974, all three services had TTA's operating and either 
managing or monitoring all OT&E programs. The Army and the Air 
Force agencies were and continue to be quite similar in size and 
mission. The Navy had a test agency in existence which was near¬ 
ly independent of the developer and user; through mission re¬ 
orientation and some restructuring, it assumed the full OT&E 
management responsibility. 

The costs associated with the implementation and operation 
of the TTA's arc substantial. However, due to the lack of a 




baseline for comparison, it is difficult to assess how much more 
costly the new ITA program is from the previous policy of joint 
developer/user testing. OT&E cost accounting was and remains 
very difficult because of the number of organizations involved 
in OT&E and the various types and sources of funding utilized. 

There is evidence indicating that because of the emphasis 
placed on more testing by the BRDP, Congress and OSD, the 
Services have overreacted. The pendul\im may have swung too far, 
to the point where additional testing has become non-cost effec¬ 
tive. 

As seen by a DOD Program Manager, costs resulting from the 
introduction of ITA's are more than just dollar oriented. As 
significant are the losses in control of his program, a usurping 
of his implied authority as the indivi(?”.al charged with total 
program management responsibility. 
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SECTION VI 


CONCLUSIONS 

Through the establisliment of ITA's, the DOD systems acquisi¬ 
tion process has incurred an abrupt change to the manner in which 
systems OT&E is managed, conducted and reported. With any abrupt 
change in policy, there is often a tendency on the part of the 
directing agency to overemphasize the change and the reasons for 
it. This overemphasis in turn leads to an overreaction by the 
organizations upon which the policy was directed. 

It is quite possible that such a cause and effect relation¬ 
ship existed with the direction for more OT&E and the implementa¬ 
tion of ITA's. Evidence supports the fact that the new OT&E 
policy has resulted in higher RDT&E costs. However, complete 
substantiation of this is only possible by first improving T&E 
cost accounting systems within all Services. 

Last, but not least, the ITA concept has been costly to DOD 
program managers in terms of proper program management control. 
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SECTION VII 


RECOMMENDATIONS 

Three significant T&E issues have emerged as a result of 
this research. The issues are interdependent, as are the recom¬ 
mendations for their resolution. 

The first issue is the inadequacy of the military services' 
T&E cost accounting procedures. This factor also makes the T&E 
budgeting processes themselves suspect. It i.s recommended that 
the services take a "hard look" at their T&E budgeting and cost 
accounting procedures, idt'ntify the weaknesses, and institute 
improved financial control. 

The second issue is that of possible overtesting during 
system OT&E. Tne Services should analyze their current OT&E 
planning at all levels to insure that only the minimum necessary 
is being planned and accomplished. 

The third and major issue is the need for ITA's. Recommend 
that OSD and the Services reassess and reverify the raquirement 
f independent testing as it is currently conducted. This 
assessment could be based upon those system-oriented fcccors that 
the DOD Program Manager considers daily; namely, cost, schedul';, 
and performance. 
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